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ABSTRACT 
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the Bernard van Leer Foundation and attended by representatives of projects 
and organizations involved in promoting early childhood development. The goal 
of the workshop was to help participants design and carry out an evaluation 
of their own project without having recourse to an outside expert. Topics 
covered in the report include: (1) identifying barriers to evaluation; (2) 

definitions of evaluation; (3) reasons for evaluation; (4) identifying the 
aims of evaluation; (5) focusing the evaluation; (6) ways of finding 
information; (7) generating indicators; and (8) implementing an evaluation. 
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particularly useful to project staff and resource 
centres, and is of great interest to project 
directors, policy makers, and academics. 
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Background 

This paper is based on a workshop held in Tel 
Aviv, Israel, in May 1997. The one day workshop, 
facilitated by the author, was carried out on 
behalf of the National Council for the Child 
with the support of the Bernard van Leer 
Foundation. The workshop attracted 18 
participants representing 1 1 different projects 
and organisations involved, in various ways, in 
promoting early childhood development. 

The aim of the workshop 

The main aim of the workshop was to help 
participants to design and carry out an evaluation 
of their own project without having recourse to 
an outside ‘expert’. 

The workshop aimed to ‘keep it simple’, to deal 
with evaluation in a very practical and down to 
earth way. 

The content of the workshop 

The workshop included selected aspects of 
evaluation theory which were intended to help 
participants to achieve control over the design 
of an evaluation, and combined these with a 
practical task-centred approach which allowed 
participants to focus on their own projects. 

This report has sought faithfully to record the 
proceedings of the workshop in the chronological 
order in which they occurred except in two 
respects: 



m section Who is the evaluation for? 

(page 10) was not dealt with in the 
workshop because of lack of time but it 
is included here for completeness; 

^ a few points in this report are out of 
chronological order because this adds to 
the coherence of the report. 

The purpose of this report 

This report records the workshop for three reasons: 

m to help participants reinforce any learning 
gains they have made; 

m in the hope that the content of the workshop 
may be of interest to people who were not 
present; 

m in the hope that the format of the workshop 
may be of interest to people who wish to 
promote evaluation. 
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Evaluation does not happen as often 
as it should 

Evaluation does not take place as often as it 
should. Some projects do not evaluate their 
work at all, while others evaluate only certain 
aspects of their work or else they may evaluate 
too infrequently. Participants in the workshop 
were invited to give reasons why this happened. 

Barriers to evaluation 

The following possible barriers to evaluation 
were elicited during a full group discussion. 

Fear 

m of an evaluation leading to change and 
disruption; 

m of raising issues which are currentiy dormant; 

m , of ‘rocking the boat’; or waking the elephants’; 

m of negative findings; 

^ of having one’s mistakes made public - of 
being judged and found wanting; 



Not valuing evaluation 

m evaluation is not necessary because ‘we 
know it all already’; 

m evaluation is not always useful because 
evaluation findings are not always 
implemented; 

m the benefits of evaluation may not be 
easily perceived, therefore why do it. 

Lack of skills and knowledge 

m a lack of knowledge about what to evaluate, 
which can be made worse by the fact that 
texts which deal with evaluation theory are 
often complicated and this does nothing to 
encourage evaluation practice; 

m a lack of skills needed to conduct an 
evaluation. 

It is important to identify why evaluation does 

not take place as often as it should so that the 

reasons can be addressed. 



^ in the worst case, of losing one’s job. 

Costs 

evaluation costs money; 
evaluation costs time. 
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What for? What purpose should evaluation serve? 



Monitoring and evaluation 

The workshop noted that there was frequent 
confusion between monitoring and evaluation. 
The following simple distinction was drawn 
between the two: 

. . . 'monitoring^ allows us to describe what we 
are doing. And 'evaluation, which involved an 
element of analysis', is about learning from, 
and understanding, that experience. 

Van derEyken W. 1992 

A simple definition 

A straightforward dictionary definition of 
evaluation is: 

To put a value on something. 

This definition is important in that it implies a 
positive aspect to evaluation: it seeks to put a 
value on activities rather than to devalue them. 



Whose value? Whose value should be put on 
something? 

What kind of value? There are different ways of 
'measuring value' for example by counting things 
(quantitative methods) or by assessing the quality 
of things (qualitative methods). 

In discussion, participants added another two 
questions: 

Who should evaluate? 

When should evaluation take place? 

Complexities of definition 

In fact, it is easy to generate more and more 
questions but this process is ultimately self 
defeating because it creates unhelpful complexities. 
The facilitator gave the example of a training 
course he had attended where 62 different types 



Towards a more 
useful definition 

It is obviously important to 
go beyond this 
definition however. 

The facilitator suggested 
three important questions 
which arose from this 
simple definition: 





Illustration: by Pia Mobouck is taken from Introducing Evaluation by Willem van der Eyken (1992) 
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of evaluation were presented. This was far too 
many to be useful and was guaranteed to add 
to any existing feelings of confusion. 

A better approach was to identify a small number 
of important ‘dimensions^ to evaluation which 
participants could consider. These dimensions 
to evaluation were linked to the six questions 
dealt with above. The term ‘dimension’ is used 
to include both a particular type of approach to 
evaluation and a particular purpose which 
evaluation may serve. 

Six important dimensions to 
evaluation 

The following six dimensions to evaluation 
were considered. 

What for? 

Goal centred, and goal free 
evaluations. 

The summative and formative aims 
of evaluation. 

Whose value? 

Pluralistic and single perspective 
evaluations. 

What kind of value? 

Qualitative and quantitative research 
methods. 

Who should evaluate? 

Internal or external evaluations. 



When should evaluation take place? 
Ongoing evaluation or one-off 
evaluation (usually at the end of a 
project). 

Each of these six dimensions is dealt with 
separately below. In the case of internal and 
external evaluations and one-off or ongoing 
evaluations, participants had useful direct 
experiences to contribute and, although these 
were in fact dealt with towards the end of the 
workshop, they fit better into this section of 
the report. 

Goal centred or goal free? 

Goal centred, or ‘impact’ evaluations are used 
when the aim is to assess ‘success’. Various 
types of ‘goals’ may be used in evaluations. 
Most frequently, a project will be concerned 
to discover how far it has achieved its aims. 
Another common goal is cost effectiveness, 
and funders in particular may wish to prioritise 
finding out how efficiently money has been 
spent. There are many other possible types of 
goal however, including finding out how the 
work of a project compares to an accepted 
measure of good practice (e.g. are we providing 
‘good quality’ childcare?). 

Goal centred evaluations are particularly 
useful when the aim is to improve project 
practice. A goal centred evaluation will 
therefore also usually be a ‘formative’ evaluation 
(see next page). 
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What is evaluation? 




Goal free, or 'process’ evaluations are most 
useful when the aim is to understand what is 
happening in a project. They do not aim to 
measure 'success’ but rather to illuminate 
practice. 



how far project aims have been met. In fact 
there are many different kinds of goals against 
which success may be assessed, not all of them 
established by the project, and even 'goal free’ 
evaluations are potentially valid, at least in theory. 



Goal free evaluations may be most useful when 
the process in a project is complicated and 
needs to be understood, or when it is innovative 
and may be of interest beyond the project, or 
when the process is clearly not working but the 
reasons are not known. 

Although there are situations where a goal free 
evaluation would be appropriate, in practice 
most evaluations are related to some kind of 
goal or objective. The main purpose of drawing 
the distinction between goal centred and goal 
free evaluations is to challenge the idea that an 
evaluation must automatically focus on seeing 



Summative or formative evaluations? 

A summative evaluation seeks to describe 
(summarise). This may seem somewhat futile 
at first but in fact a summative evaluation can 
be very useful to people who want to be kept in 
touch with what is happening in a project, and 
to be enabled to make their own analysis and to 
form their own conclusions. Audiences who 
may welcome a summative evaluation include 
projects involved in similar activities, project 
management, and funding agencies. 

A formative evaluation aims to change project 
practice. It will provide some background and 
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description, will offer an analysis and will 
recommend future courses of action. Formative 
evaluations are of obvious benefit to everyone 
with an interest in improving the work of a project. 

In practice, most evaluations will aim to be 
both summative, to describe the project, and 
formative, to improve project practice. 

Pluralistic and single perspective 
evaluations 

A pluralistic evaluation will aim to present 
more than one perspective. Among the 
perspectives which an evaluation may want to 
consider are: funders, managers, staff, parents, 
children, professionals and paraprofessionals, 
other projects, and academics. All of these 
groups have a stake in a project and can be 
called ‘stakeholders'. 

Unless there is a good reason specifically to 
focus on one perspective (for example to 
explore the satisfaction of parents with services 
provided to their children), it is important to 
present the views of several groups on impor- 
tant issues because what constitutes ‘success’ 
will vary, and may in fact be very different, 
depending upon who you ask. 

The fact that success depends upon who you ask 
was illustrated in the workshop by a cartoon. 

This showed two cats reading a newspaper 
headline which said 

'Woman finds dead rat in a packet of fried 
chicken with one cat commenting to the other 
'Some people have all the luck!' 



The essential point here is that ‘success’ is 
subjective and that the value placed upon a 
particular event, will vary according to the 
perspective of the observer. 

It is extremely important therefore that 
evaluations seek to present the perspective 
of more than one ‘stakeholder’ group. 

Qualitative and quantitative research 
methods 

Any evaluation in the field of early childhood 
development will be able to use some quantitative 
measures but it will also need to obtain reliable 
qualitative data. This is an important point 
which is dealt with more fully in the section 
‘Finding valid answers’ (page 14). 

Internal and external evaluations 

An internal evaluation will usually be carried 
out by a member of project staff, or sometimes, 
by volunteers. An internal evaluation should 
cost less in money but more in time. It will 
benefit from the evaluators’ existing knowledge 
of the project but they may find it difficult to 
detach themselves sufficiently to see the ‘big 
picture’. It will be easier for a project to control 
an internal evaluation and to focus it on the 
questions the project thinks is important. An 
internal evaluation may not however have as 
much credibility as a more independent external 
evaluation with outside bodies such as funding 
agencies. 

In many ways the advantages and disadvantages 
of an external evaluator are the exact opposite 




What is evaluation? 



of the above. These are the points made by a 
project which has a great deal of experience of 
working with external evaluators: 

External evaluators are particularly useful when 
there is a perceived need for 'objectivity'. Another 
advantage is that an external evaluator will take 
less time out of project work. There is however an 
obvious monetary cost in engaging an external 
evaluator. There is a particular difficulty with 
'academic' evaluators in that they tend to focus 
on 'measurable' outcomes which may not be what 
the project is most concerned with. It can also be 
difficult to get an external evaluator to focus on 
the project's objectives. 

One-off evaluations or ongoing 
evaluations 

One-off evaluations are usually retrospective 
evaluations which are carried out towards the 
end of a project. 

An ongoing evaluation will start at the same 
tirne, or even before a project and continue 
throughout the project’s life. It allows for 
appropriate recording and monitoring systems to 
be put in place early in project life which ensures 
that information is gathered; it allows for feed- 
back to be provided to the project at regular 
intervals; and the evaluation itself can change 
its focus in response to changes in the project. 

Ongoing evaluations present many other 
advantages. This is the perspective of an evaluator 
engaged in an ongoing evaluation who 
participated in the workshop: 



The evaluator reports directly to the funding 
body and the main links with the project are 
administrative. The evaluation is therefore fairly 
independent. 

The evaluation has a certain amount of days 
available to it each year. This very useful because 
it allows forward planning. It is possible to work 
less at quiet times and to work more or less full 
time when necessaryy for example around the 
time of preparing an annual report. 

Because the evaluator is there all the time it is 
possible to be flexible, for example to be available 
to observe special events which are felt to be 
important. 

An ongoing evaluation can produce regular 
reports. These provide formal feedback to 
management teams and participants which helps 
the project to respond quickly to development and 
to make changes without interrupting the devel- 
opment of the project. Informal feedback is also 
possible even between reports which also assists 
the project in reviewing and changing what it 
does. 

It is clear that ongoing evaluations are in many 
ways preferable to one-off evaluations. 
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What is the evaluation intended to 
achieve? 

Workshop participants generated some 
questions which suggested why they might wish 
to evaluate their own project: 

m Is the project meeting its goals? 

m Is the project cost effective? 

m Is a pilot project potentially replicable? 

m Are the project’s goals appropriate or 
should they be changed? 

m How does the project use its resources? 

m Is the time scale for the project appropriate? 

m Is the project appropriate to the special 
needs or characteristics of the community 
it works in? 

m Are the right participants involved in the 
project? 

m Does the project have a distinct role or 
should it join with other projects involved 
in similar work? 

All these questions provide reasons for evaluation 
and give an indication of the type of benefits 
which may result from evaluation. 



Who is the evaluation for? 

It is also important to remember however, that 
there is a range of possible audiences for an 
evaluation, each of which will be interested 
for different reasons. Children, parents, 
paraprofessionals, project staff, managers, 
funders, other similar project, academics will 
want different things from an evaluation. 

This means that it is important to decide who 
the main audiences are, and to ensure that the 
final evaluation report is written in a style and 
language which is appropriate to the audience. 
Different audiences will require different styles 
and possibly different products. For example, 
professionals will be familiar with reports 
written in technical language but parents of 
children may need more everyday language; 
project staff who are directly concerned with a 
project may be happy to read a long report 
while external professionals may prefer a shorter 
document; a report for internal project use can 
be produced on the office photocopying machine 
but one which aims to impress should perhaps 
be more professionally produced. Also of course, 
a report is only one possible product of an 
evaluation; there are circumstances where 
leaflets or videos for example might be 
appropriate. 

The key point here is that, having identified 
the purpose of the evaluation and the main 
audiences, decisions have to be taken about 
the ‘length, look and language’ of the report. 
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Avoiding trying to do too much 

Many evaluations try to do too much, it is 
important to decide a specific aim for an 
evaluation and not to be over ambitious. 

One phrase which returned time and again 
in the workshop was: 

Keep it simple. 

Three questions to help focus an 
evaluation 

In the workshop participants looked at their 
own project and decided what they wanted to 
evaluate. 

Participants were asked to bear three questions 
in mind to help them choose a focus for their 
evaluation: 

m What am I proud of about my project? 



m I want to measure if my project is really 
meeting its aims, in particular its main aim 
m of helping children to mature and develop? 

m Does our ‘high tec’ computer training 
project fit the local population; is it 
acceptable to them? 

m An evaluation of my project should aim to 
measure its efficiency and productivity, the 
cost benefit of our work . . . 

m Are we using our resources appropriately 
(resources of finance, staff, time, and 
services available from outside agencies? 

The range of responses reinforces the fact that 
evaluation cannot do all things for every project. 

An early stage in an evaluation process is 
therefore deciding what is the most important 
thing you want evaluation to do for you. 



n What am I worried about in my project? 



m What is unique about my project? 

A wide range of possible aims 

Responses indicate a great variety of project, 
of social and cultural settings and of priorities. 

□ Is my project promoting the real integration 
on an equal basis of Ethiopian immigrants 
into Israeli society? 
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Asking the right questions 

Even when the main focus has been chosen a 
risk remains of trying to make an evaluation do 
too much. 



It is important to assert control over the 
evaluation and choose what are the most 
important specific questions you want 
answered about your work. 



Evaluation begins and ends with questions. 
Asking the right questions is the most difficult 
part of the whole process. 

(Van derEyken 1992 ) 



The workshop highlighted the 
importance of asking ‘the right 
questions* 



Achieving a balance 

The right questions are 
questions to which 
there are clear 
answers. In other 
words they must 
be specific. 



however. If they are too specific, they lose their 
generality and therefore become less useful. 

Unfortunately, no one has ever written a guide 
to asking the right questions for every project 
in every setting and for every priority which 
exists. This, therefore is the point where it is 
useful to remember the saying that ‘evaluation 
is an art, not a science*. No one can tell you 
when a question is right. You have to think it 
out for yourself, then try out the questions by 
piloting them on small numbers of people. 

Participants took the broad aim they had 
defined for their own evaluation as the starting 
point for defining their own questions and 
were asked: 






W 






£ 






They must 
be general 
enough. 



also 




Focusing the evaluation 



What are the four most important questions you 

want the evaluation to answer about your 

project, and why do you want to ask these 

questions? 

Two examples from the workshop follow. 

Example one: 

Main aim: 

m An evaluation of my project should aim to 
check whether we are achieving our goal 
of developing and enriching children s 
abilities in early childhood. 

Specific questions: 

□ Is the programme supporting development 
in the target population (children aged 0 
to 6)? 

m Are the materials used in the programmes 
appropriate to the local population? 

□ Are local and national staff efficient and 
effective? 

□ Are we using our budget appropriately? 

Example two: 

Main aim: 

□ An evaluation of my project should aim to 
evaluate how far the project has achieved 
its goals of improving the safety of children. 

Specific questions: 

□ Is the programme raising awareness of safety? 



m Is the programme enjoyable? 

e Are the children learning at least one new 
thing which will improve their safety? 

Having made a first attempt at asking the right 
questions, it is always necessary to define terms 
further, and to rephrase the questions. Both 
examples above however, illustrate the first step 
in generating more specific questions from a 
general aim. 
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Three types of research 

Devising detailed research methods was beyond 
the scope of this short workshop, and 
approaches to research were in fact simplified 
into three basic ways of finding information: 

® asking people questions; or 

® watching people; or 

® reading things. 

Finding valid answers 

It will be necessary to use qualitative as well as 
quantitative research methods. The workshop 
therefore looked specifically at the question, 
how do we ensure that the finding of qualitative 
research are valid? 



This part of the workshop started with the 
premise that there is a widespread tendency to 
value quantitative above qualitative research 
findings, to value objectivity above subjectivity 
and to use terms like ‘scientific' to mean the 
same thing as valid. 

There are good reasons to dispute this view. 
Three reasons are: 



deduce that if people keep attending they 
will be generally satisfied but we cannot 
use these or any other figures to assess 
what people have learnt and whether they 
are putting these lessons into practice; 

® the definition of success depends upon 
who you ask. A child may value a stimulating 
environment; a parent may place more 
emphasis on a safe environment; while 
a funder may consider that cost effective 
ness is an important criterion for success; 

^ discussion in the workshop added that the 
definition of success also depends upon 
who does the asking. An evaluator will 
bring to their work their own particular 
values, and perceptions based upon their 
own experience. This is particularly 
important in a culturally diverse society 
such as Israel. 

Triangulation of sources and methods 

The workshop highlighted ‘Triangulation’ of 
method and of source as a means of promoting 
valid findings in qualitative research. Very simply, 
triangulation means using more than one 
source (ideally at least three sources) of 
information, and using a variety of research 
methods. 



a lot of what we want to find out cannot 
be fully quantified. For example, we can 
count how many people attend a training 
course, we can use attendance figures to 



When more than one source of information is 
used it is important to identify whose truth is 
being reflected in any part of the evaluation 
report. 
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Finding answers 



If all sources are in general agreement on an issue 
then the evaluation can be reasonably sure of its 
analysis. If the sources have different views on an 
issue however the evaluation should express 
their varying perceptions before coming to a 
necessarily more tentative conclusion of its own. 



evaluation report with the various groups of 
people who have participated in the research, 
so that amendments can be made before the 
final report is produced. This process can be 
called Validation through consultation’ 
(Marsden D & Oakley P (Eds) 1990). 



Validation through consultation 

This point was raised later in the workshop but 
is included here for better coherence. 

An additional way of ensuring that qualitative 
research findings are valid is to discuss the draft 




It is however important to protect the 
independence of the evaluation at this stage as 
it is possible that the evaluator will have 
reached some conclusions which they do not 
wish to change but which some parties will 
disagree with. One way of preserving the 

independence of the evaluation 
while respecting varying views 
is formally to agree with 
participants that, in the event of 
disagreement the evaluator retains 
the right to decide upon the final 
content of the report but that 
participants also have the right 
to have their views recorded in 
an appendix to the evaluation 
report. 

Who has the answers? 

In the next part of the workshop, 
participants were invited to 
identify where the information to 
answer their questions could be 
found. They were asked to address 
the following questions in relation 
to one of the more specific 
questions generated in the 
previous part of the workshop. 
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In order to find the answers to each of your 
questions: 



These lists illustrate that there is a wide range 
of sources of information, and more sources 
could of course be added. 



m Who can you ask? 



ea Who or what can you observe? 
m What written material exists? 



Between them workshop participants very 
quickly generated the following possible 
sources of information for the questions they 
were asking: 

ASK: 

parents, paraprofessionals, other 
professionals, project staff, project 
management, national management, 
kindergarten teachers, school teachers, 
trainers, visitors to the project etc. 

OBSERVE: 

children in project activities, parents in 
project activities and interacting with 
children, staff in meetings and conducting 
training and conducting activities, etc 

READ: 

project resumes, project reports, 
certificates of qualification (re training 
courses), reports of training courses, job 
descriptions, programmes of work, records 
of staff meetings, attendance figures of 
various kinds, finance records, etc. 
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What is an indicator? 

In this part of the workshop participants gener- 
ated ‘indicators’ in relation to the specific 
question for which they had identified the 
likely sources of information. 

There was some discussion about what was 
meant by ‘an indicator’. One distinction is that 
quantitative research looks for measures, that 
is, findings which can be counted. Qualitative 
data however looks for types of behaviour, or 
events, or widely held opinions which indicate 
that something is taking place. 

Setting indicators, like defining specific 
questions, is an art rather than a science, and 
there is no single way of arriving at appropriate 
indicators. 

Using focus groups to establish locally 
important indicators 

One approach which was suggested in the 
workshop is to hold focus groups to discuss the 
questions being asked by the evaluation. These 
focus groups, which will be composed of 
different types of stakeholders, will generate 
data (which will in itself be useful for the 
evaluation) which may be used to identify^ 
useful indicators. 



For example, if a project is asking ‘how successfully 
have we promoted child development’, focus 
groups involving perhaps parents of children, 
and project staff could be asked what signs they 
have seen that children are being helped to 
develop. The signs they generate may then be 
taken as indicators of child development. 

Structured questionnaires or observation 
frameworks may then be developed from these 
indicators. The structured questionnaires may 
be administered to a larger sample of parents, 
staff and perhaps other professionals who are 
familiar with the project, while the observation 
frameworks may be used to observe children 
taking part in project activities. 

The process may be summarised as follows: 

Step 1: Set up focus groups to discuss the 
main questions you want the 
evaluation to address. Each group is 
composed of people you have 
identified as having information on 
the question. These focus groups 
generate qualitative data which is 
useful to the evaluation. The main 
points generated by the focus groups 
can be used as a basis for more 
structured questionnaires and 
observation frameworks. 

Step 2: Prepare and pilot questionnaires and 
observation frameworks based upon 
the findings of the focus groups. 
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Step 3: Administer the questionnaires and 

conduct the observations (at the same 
time you can be collecting all useful 
written information from project 
records). 

Step 4: Analyse the findings. 

A further advantage of this approach is that it 
recognises that definitions of ‘success’, in this 
case in child development, may well vary for 
example across different communities, and in 
different ethnic groups. The focus groups, 
being composed of people with a local 
perspective will necessarily generate locally 
important and culturally appropriate 
indicators. This constitutes an additional 
recommendation for this approach in a diverse 
country such as Israel. 

Participants generated their own indicators for 
one of the questions they had asked about their 
project. One example of indicators to evaluate 
the success of a project which aimed to involve 
mothers in training their children in computer 
skills was: 

m How clearly does the mother explain to the 
child what is expected? 

E How clearly does the mother demonstrate 
to the child what is expected? 

□ How does the mother correct the child 
when he makes a mistake? 



@ How many times does the mother ask for 
help? 

m How comfortable does the mother feel 
when she is teaching her child computer 
skills? 

m How happy are both mother and child? 

m Does the child perform the range of tasks 
required to operate the computer correcdy? 

The obvious indicator of success is listed last of 
all and the other questions look at additional 
indicators. The questions take into account the 
feelings of the mother and child (indicating for 
example how likely they are to remain involved 
in the longer term), as well as the ability of the 
mother to train her child (indicating the 
success of the project in training the mother). 

One question ‘How many times does the 
mother ask for help’ is interesting because the 
same response would probably indicate success 
at one stage in the training process but lack of 
success at another. For example if the mother 
asks for help often, is this an indicator of 
success (the mother is comfortable enough to 
ask for help) or not (the mother has not 
learnt). Quite possibly asking for help frequendy 
would be an indicator of success in the early 
stages of her involvement with the project but 
would indicate lack of success if she still had to 
ask for help often after being involved with the 
project for a long time. 
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Generating indicators: what are you looking for and how do you know when you have found it ? 




The questions are also clearly suitable for 
different research methods. Some questions 
can be asked directly (The feelings of the 
mother), while others will require observation 
(how well the child performs the task, how 
well the mother trains the child). 

These questions illustrate how it is possible to 
generate some good, versatile indicators very 
quickly, even in a workshop situation. 

An alternative approach to generating 
indicators 

Using focus groups alone is a very simple way 
of generating indicators but other more 
sophisticated approaches are also possible. One 
project represented at the workshop outlined 
its usual approach. 

Step 1: begin with non structured interviews 
with key people, unstructured 
observations and reading easily 
available project records. 

Step 2: analyse all the material you have 

gathered from the above and use this 
to generate indicators which can be 
used to produce structured 
questionnaires and structured 
observations. 

Step 3: administer the structured 

questionnaires and conduct the 
structured observations. 



Step 4: identify any gaps in the research which 
need to be filled with additional 
interviews and conduct these. 

Step 5: analyse all the data. 

This project noted that at the end of step one 
they usually found they had quite a ‘rosy’ 
picture and that the questionnaires in step 
three should be designed to be quite probing in 
order to arrive at the full, not just the rosy, 
picture! 

This example reinforces the point that there is 
no single way of deciding upon indicators. 
There are many ways, all of which may work 
and it is up to each project to select or define 
for themselves an appropriate approach. 

Ensuring that locally important 
indicators remain universally valid 

The workshop also recognised however that 
there are certain fundamental factors which do 
not vary across communities and cultures, 
principally those to do with meeting basic 
needs. It will therefore also be important to 
refer to authoritative sources, not to generate 
indicators, but rather to ensure that locally 
generated indicators are within the broad 
framework of what is widely recognised as 
good practice. 
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Resources required for evaluation 

The next section dealt briefly with 
implementing an evaluation. 



The facilitator introduced some important 
resources required by an evaluation. 

Records: 

Boxes, one for each main question, are the 
very simplest form of recordingsystem to 
hold information which may be useful to 
an evaluation! It is, of course, possible to 
devise more elaborate recording systems! 

Time: One person (a designated evaluator) or 

more than one person (an evaluation 
working group) who has time to 
devote to carrying out the evaluation. 



An evaluation action plan 

The facilitator outlined five phases of an 
evaluation action plan and some of the issues 
likely to be important in each phase. 

Five phases of an evaluation action plan. 

Phase 1: Preparation and planning 

Important issues: 

m The issues we have addressed today 

m Do any new recording systems need 
to be put in place? 

m Does anyone need any more training 
to carry out the evaluation? 

M Are we being realistic about the time this 
will take (it is easy to be over optimistic)? 



A support and management system: 

This is necessary to assist the evaluator/ 
evaluation working group to review 
progress with the evaluation, amend 
plans, provide guidance and support, 
and when necessary to troubleshoot. 

In practice this may take the form of 
an evaluation management or advisory 
group which meets regularly with the 
evaluator. 



m Designing and piloting the research 
instruments. 

Phase 2: Conducting the research 

Important issues: 

m Is the evaluation keeping to schedule? 

m Are our methods working as we 

planned if not what do we do about it? 



An action plan: 

An action plan must state clearly, what 
is going to be done, who is going to do 
it, and when it will be done. 



Phase 3: Preparing the report 

Important issues: 

□ Who sees the draft report? 
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o Who has the right to amend it? 

Phase 4: Printing and circulation 

Important issues: 

□ Who receives the final report? 

m Is there a need for different reports for 
different audiences ? 

Phase 5: Evaluating the evaluation 

Important issues: 

m What worked well, what did not work 
so well, what will we do differently 
next time? 



Workshop participants then developed the 
discussion concerning record keeping and 
finally funding for evaluation. 

What kind of record keeping? 

On a very basic level it is important to have 
somewhere for workers to put information 
about a particular subject, in order to make sure 
information is not lost and to encourage people 
to collect the information. One box or box file 
for each main question being asked by the 
evaluation is a very simple way of achieving this. 

A form which workers have to fill in and which 
asks the questions the evaluation wants to ask 
has several advantages: it focuses records on 
relevant material; it makes analysis much easier 
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because responses are standardised; and it 
removes the uncertainty workers often feel 
about what is or is not relevant information for 
an evaluation. 

A ‘timeline’ offers an alternative way of structuring 
information. Workers are asked simply to note 
when ‘an important’ event occurs in the project. 
The evaluator can then interview staff about 
why these events were important. 

Alternative forms of record keeping which were 
used by different participants were: photographs 
and videos; examples of children’s work which 
could show how a child’s skills and perception 
develops over time; and a book in which 
visitors to the project are asked to record their 
observations thereby capturing a range of 
external perspectives. 

Funding for evaluation 

Adequate funding is of course an essential 
resource for an evaluation. A crucial point 
made in the workshop was that the best way to 
ensure this finance is to build in funding for 
evaluation in grant applications. 

By doing this a project will also demonstrate a 
commitment to examining and improving its 
own work which may impress potential funders. 

Where however, it is felt that funders would not 
be prepared to support an evaluation some costs 
of evaluation (stationery, communications etc) 
could be legitimately included in ‘core costs’ in 
a funding application. 
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In the meantime it is clear that the workshop 
brought together representatives of 1 1 projects, 
organisations or networks and allowed them to 
focus purely on evaluation for a full day. 

Participants brought with them a great deal of 
energy, a wide range of experience, and strong 
analytical skills which enriched the workshop 
and helped it almost completely achieve its 
immediate aims of completing what was an 
ambitious programme. It appears too that the 
day was in the main characterised by enthusiasm 
and active participation. 

Only time will tell however whether the 
workshop will achieve its overall aim of helping 
participants to design and carry out an 
evaluation of their own project without having 
recourse to an outside 'expert’ 

The evaluation of the workshop itself will 
provide some indication of what may happen 
but the 'hard’ indicator will be whether those 
who attended evaluate their work more 
enthusiastically, and more often in the future. 

It will be interesting to review the impact of the 
workshop a year or so from now. 
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About the Foundation 

The mission of the Bernard van Leer Foundation is to 
enhance opportunities for children 0-7 years, growing 
up in circumstances of social and economic disadvantage. 
The objective is to develop children's innate potential 
to the greatest extent possible. We concentrate on 
children 0-7 years because research findings have 
demonstrated that interventions in the early years of 
childhood are most effective in yielding lasting benefits 
to children and society. 

We accomplish our mission through two 
interconnected strategies: 
d a grant-making programme in 40 countries 
aimed at developing contextually appropriate 
approaches to early childhood care and 
development; and 

d the sharing of knowledge and know-how in 
the domain of early childhood development 
that primarily draws on the experiences 
generated by the projects that the Foundation . 
supports, with the aim of informing and 
influencing policy and practice. 




The Bernard van Leer Foundation is a private 
foundation created in 1949 that is based in the 
Netherlands and operates internationally. The 
Foundation's income is derived from the bequest 
of Bernard van Leer, a Dutch industrialist and 
philanthropist who, in 1919, founded an industrial 
and consumer packaging company that was to 
become in later years the Royal Packaging Industries 
Van Leer, currently Huhtamaki Van Leer, a limited 
company operating in 55 countries worldwide. 
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Working Papers in Early Childhood Development form 
a series of background documents drawn from field 
experience that presents relevant findings and 
reflections on 'work in progress'. The series therefore 
acts primarily as a forum for the exchange of ideas. 

The papers arise directly out of field work, evaluations 
and training experiences. Some of these are from 
the world wide programme supported by the Bernard 
van Leer Foundation. All the papers are aimed at 
addressing issues relating to the field of early 
childhood care and development. 

The findings, interpretations, conclusions, and views 
expressed in this series are those of the authors and 
do not necessarily reflect the views or policies of the 
Bernard van Leer Foundation. 

Copyright is held by the Foundation. Unless 
otherwise stated, however, papers may be quoted 
and photocopied for non-commercial purposes 
without prior permission. Citations should be given 
in full, giving the Foundation as source. 

Contributions 

The Foundation solicits contributions to this series. 
They should be drawn from field experience and be 
a maximum of 15,000 words. Contributions can be 
addressed to Joanna Bouma, Department of 
Programme Documentation and Communication at 
the address given above. 
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